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ABSTRACT 

The pilot Job Effectiveness Training (JET) workplace 
basic skills program, developed by Canada's Alberta Vocational 
College (AVC), Edmonton, for Stelco Steel during 1989-90, was 
evaluated in terms of impacts or changes from the perspective of the 
four major stakeholder groups: the students (12 Stelco employees); 
the employers (Stelco management) a the institution (administration of 
AVC); and the government funding agencies. Impacts were determined 
through structured interviews, with an additional questionnaire for 
stutients that replicated portions of the post-program evaluation. The 
study found at least 54 positive impacts and only 3 negative impacts; 
of the impacts, 30 were planned and 24 were unplanned. Student 
impacts achieved included a 20 percent increase in skill levels, 
retention of increased skills, use of skills learned, greater 
self-confidence, and a positive attitude toward education. For the 
employer, JET provided access to a system for upgrading its work 
force, upgrading some employees, improved communication, and 
awareness of the importance of basic skills to job performance. AVC 
outcomes included development of a generic workplace basic skills 
curriculum, enhanced reputation, experience with nontraditional 
delivery, and development of a model for interaction with business 
and industry. Funders" outcomes were development of an innovative 
literacy project and positive publicity. The study concluded that the 
project was highly successful for stakeholders and that it was cost 
effective. (KC) 
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Executive Summary 



The purpose of this study was to evaluate a pilot workplace 
basic skills program, specifically the JET (Job Effectiveness 
Training) curriculum developed by Alberta Vocational College 
Edmonton for Stelco Steel during 1989-90. The evaj. aation was 
conducted in terms of impacts or changes from the perspective of 
the four major stakeholder groups: the students (12 Stelco 
employees) ; the employers (Stelco management) ; the educational 
institution (administration of AVC Edmonton) ; and the funding 
agencies (representatives of the government departments who 
participate in the National Literacy Partnership Strategy) . 
Impacts wei.e determined through structured interviews, with an 
additional questionnaire for the students which replicated 
portions of the post-program evaluation. The study concluded 
that there were at least 54 positive impacts and only 3 potential 
negative impacts; of the impacts, 30 were planned and 24 were 
unanticipated. It has been concluded that this program was 
highly effective from the point of view of all stakeholders. In 
addition, it may be concluded that the program was highly cost- 
effective, although the process for determining this conclusion 
is tentative at best. This study provides a large number of 
specific recommendations for workplace basin skills programming 
from all the stakeholders. It is recommended that more impact 
studies be conducted of workplace basic skills programs in order 
*;o provide a basis for comparison. 



The Stelco JET program under study 

With a grant from the National Literacy Partnership 
Strategy, AVC Edmonton developed a workplace basic skills 
curriculum and piloted it at Stelco Steel, Edmonton Works from 
November 1989 to April 1990. The Job Effectiveness Training 
(JET) program is a generic curriculum intended to address basic 
skills in the workplace by serving as a basis on which to 
customize an educational upgrading program for a specific 
business or industry. AVC Edmonton planned and hosted the pilot 
program for Stelco, providing the curriculum development, 
administrative, instructional and evaluative services. Stelco 
Steel participated by recruiting and rewarding the students, 
availing the instructor of relevant teaching materials and 
topics, providing guest speakers and strong moral support for the 
program . 

The objectives of the JET program were to train Stelco 
employees to cope with work-related reading, writing and numeracy 
requirements; to improve spelling of work-related terminology; to 
improve knowledge of safety regulations, personnel and union 
information, and the company; to develop learning strategies to 
enhance the future learning potential of each trainee; to 
generate improved knowledge and skill related to computer 
literacy, the metric system, interpersonal relations, and more 
advanced reading skills. Instructional materials and procedures 



in the JET program were based on tnose found specifically in the 
Stelco Steel environment. 

Twelve employees of Stelco Steel became voluntary students 
attending classes at AVC Edmonton or ' aeir days off. Stelco 
paid the students, at the time of graduation, for half of the 
actual hours each participant took part at his/her regular hourly 
wage. The students came from a variety of jobs within the steel 
plant and demonstrated a variation in basic skill levels and 
interests . 



At the conclusion of the 100 hour program, quantitative and 
qualitative results included the following: 

1. over 20 % gains in objective test scores in reading, writing 
and numeracy. 

2. high levels of student satisfaction with the course, 
especially with the topics such as computer literacy, the 
metric system, the steel making process, the steel industry, 
and Stelco' s customers. 

3. gains in understanding of computers, Stelco' s operations, 
and the steel making process, as cited by Stelco 's 
supervisors who observed the changes in participating 
employees. 

4. overall improved understanding of written instructions and 
self-confidence of participants. 



The program was evaluated at the conclusion for immediate, short- 
term impacts or achievements for the students. After one year, 
there was an interest in examining the longer term impacts for 
all the stakeholder groups. 
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Overview of the impact study 



A fundamental assumption of this study was that an 
innovative educational program is implemented in order to meet 
some needs of program stakeholders. A significant need may be to 
promote change, for example improved skills and attitudes of 
participants, increased numbers of participants and partnerships 
for educational institutions, implementation of program mandate 
for agencies, an upgraded workforce for a business * Thus it was 
assumed that this Stelco JET program should have had an impact 
both immediately and over time. 

The purpose of this impact study was to determine changes or 
impacts experienced by each of the four major stakeholder groups: 
the program participants, ie. 12 Stelco employees; the employer, 
ie. Stelco Steel represented by Stelco management; the 
educational institution, ie. Alberta Vocational College Edmonton 
represented by the JET manager and senior administration; and the 
program funders, specifically representatives from Alberta Career 
Development and Employment, Alberta Advanced Education, the 
Department of Secretary of State in Edmonton, and the National 
Literacy Secretariat in Ottawa. 

The goals of the study were to assess the impact of this 
pilot program from the perspective of the stakeholders, to draw 
implications and recommendations for these and- other workplace 
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basic skills stakeholders, and to pilot the process of impact 
evaluation for workplace basic skills programs. The goals were 
all achieved. 

The process of the study was to interview representatives 
from the stakeholder groups for opinion regarding the JET 
program. In addition, the students completed a questionnaire 
which replicated relevant portions of the post-program 
evaluation. The data was analyzed, through content analysis, for 
indications of anticipated and achieved change, and of 
unanticipated but achieved change or impact. Limitations and 
delimitations to this process are acknowledged. Cost- 
effectiveness analysis was attempted and the inability to draw 
conclusions about cost-ef f ectivess were illuminated. Much of 
what stakeholders said in response to questions formed direct or 
indirect recommendations for the other stakeholders. 

Objectives, responses and observations 

To address the goals of the study, nine objectives were the 
focus of responses elicited from stakeholders, and an analysis of 
the responses lead to some generalized observations. 

The first major objective of the study was to determine the 
extent to which participation in the JET program had an impact, 



both for each individual and in a collective sense, in the work 
environment, on ability to perform work-related reading, writing, 
math and computer tasks; work-related interpersonal 
communications; work-related decision-making; and actual job 
tasks in total. It was observed that, in their own opinion, the 
majority of students had acquired improved basic skills and were 
benefitting from the JET program in performing their jobs. 

The second major objective was to determine the extent to 
which particiption in the JET program had an impact, both for 
each individual and in a collective sense, in the work 
environment, on awareness of and attitude towards corporate 
culture (Stelco goals/procedures) , union activities and 
responsibilities, opportunities for learning and participation, 
and the global context of the steel industry. It was observed by 
all the stakeholders that the JET program had an immediate and 
long-term positive impact on the participants' awareness of and 
attitude towards the corporate culture and the global context of 
their industry. Clearly, the long-term impact of this JET 
program was an increase in self-esteem and confidence in the 
participants, and an increase in willingess to learn, especially 
about work-related topics. 

A third objective was to determine the extent to which 
participation in the JET program had an impact, for individual 
students and for the group, in the non-work environment (personal 
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and social environment) , on self-perceived ability to perform 
and attitude towards essential literacy activities, leisure 
literacy activities, formal and informal learning opportunities, 
essential math activities (banking, etc.)/ computer activities, 
self-esteem, communication skills, and balance between work and 
non-work life. Only three respondents said that they planned for 
change as a result of participating in JET, yet the majority of 
participants felt that the JET program had changed their lives, 
in a positive manner, to some extent. 

A fourth objective was to determine, from the point of view 
of the employer, the extent to which offering the JET program to 
employees had an impact on productivity, specifically increased 
output; decreased accidents and/or mistakes; decreased problems 
with absenteeism, interpersonal issues, employee turnover; and 
increased commitment to the company. Stelco management was 
adamant that these measures did not and could not apply to the 
JET program. Apart from the last item, commitment to the company 
(which can not be emperically measured) , the other items had not 
figured into the initial decision to offer the JET program to 
staff. The motive had been clearly one of enhancing the basic 
skills of a static workforce with a view to increased individual 
self-esteem, learnability, and flexibility in order to affect 
long-term productivity. Stelco management was committed to the 
notion that the program must belong to the students and thus did 
not set standards of achievement or impose expectations. 
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A fifth objective was to determine, from involvement with 
the JET program, the impact on management interest in basic 
skills training for employees. Stelco management became 
confirmed advocates for JET. Every respondent could list 
positive benefits for Stelco and for the students; every 
respondent thought JET should be repeated in order to involve 
more employees. Clearly, the JET program contributed to enhanced 
interpersonal relations at Steico Steel, leading to team 
building and improved worker-management understanding. 

A sixth objective of the study was to determine, from the 
view of the educational institution, the extent to which 
developing, promoting and hosting the JET program at Stelco had 
an impact on institutional goals and activities with regard to 
priorities, resource allocation, partnerships, workplace basic 
skills programming, institutional research, unmet community 
needs, staff requirements and training, and public relations. 
AVC Edmonton, having experienced success with JET as a pilot 
community-based program, has initiated more off -site and 
customized programs. Marketing and promotion of programs has 
increased in number and importance. To address changing 
institutional priorities, additional human and financial 
resources are being allocated to community-based programs. 
The JET project was a significant "first" partnership is many 
ways and AVC Edmonton anticipates developing many more 
partnerships with business and industry, and vith the federal and 
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provincial governments. AVC intends to undertake many more 
workplace literacy programs in response to needs expressed in the 
community. The AVC JET program continues to address unmet needs 
in the immediate and wider community by providing a workplace 
basic skills model, a generic curriculum and a unique instructor 
training program. At AVC Edmonton, staff atticudes, requirements 
and training have changed substantially. Developing and 
promoting JET has generated positive public relations for AVC 
Edmonton and has established the institution as a provider of 
workplace literacy programming. 

The seventh objective of this impact study was, for the 
funding agencies individually and collectively, to determine the 
extent to which funding the JET pilot program at Stelco had an 
impact on interest in workplace basic skills programs, 
specifically increased numbers and variety of pilots, 
dissemination of successful pilot information, commitment to 
ongoing program funding, and partnerships. The eighth objective 
of the study was to determine NLPS partners' perceptions of 
success specific to agency mandate, resources expended, and 
public awareness. In short, Stelco JET was intended to 
demonstrate the viability of workplace basic skills training. 
Each representative of the NLPS partners stated unequivocably 
that the Stelco JET program had met with their expectations and 
had achieved their objectives. As a project undertaken early in 
the National Literacy Partnership Program, the Stelco JET program 
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had a role to play in helping the NLPS partners clarify the 
program, its goals and limitations. 

The last objective was to determine the strengths and 
weaknesses of this form of evaluation study with regard to 
process and outcome. The strengths are the number and variety of 
positive impacts that are surfaced through the process of 
structured interview and through involvment of all stakeholders. 
The weaknesses are related to availability and potential 
irrelevance of pre and post-test results and opinions from which 
to extrapolate impacts. Ultimately, the process of cost- 
effectiveness analysis is meaningless without comparison either 
to other programs or to standards of excellence. 

Impacts of the Stelco JET program 

The following generalizations or conclusions may be drawn 
from the observations of stakeholder responses. In total, there 
were 54 positive impacts and only 3 potential negative impacts. 
Impacts are divided into anticipated and unanticipated for each 
of the four stakeholder groups. 

For the students, the anticipated and achieved impacts were 
retention of increased basic skill levels, utilization of skills 
learned, greater self-confidence, enhanced inter-personal and 
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inter-company communications, a positive attitude towards 
continuing education lessened apprehension about learning new 
skills, increased job performance, increased motivation, and 
increased interest in the industry. The unanticipated but 
achieved impacts were changes in attitudes toward education in 
general, increasing levels of self-esteem, ability to cope with 
change (resulting from a strike and resultant downsizing) , 
confidence in self -directed learning, and confidence in the 
management of Stelco Steel. 

For the employer, the anticipated and achieved impacts 
included successful and ongoing access to a system for upgrading 
its workforce, success at upgrading some employees, development 
of an appropriate curriculum approach, confirmation that student- 
ownership of upgrading programs was appropriate, improved 
communication with graduates, and knowledge about the outcomes of 
upgrading activities. The unanticipated impacts were an 
increased awareness of the importance of basic skills to job 
performance, awareness of the skill deficits of employees, the 
ripple effect of graduates promoting upgrading to others, wider 
dissemination of the content of JET through secondary workshops, 
improved relationships between management and some union members, 
increased awareness of the services of AVC and of Alberta ' s post- 
secondary education system, increased understanding of the 
difficulty levels of communications and math relative to jobs at 
Stelco, increased efforts to incorporate plain language in 
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written and verbal information dissemination, initiation of a 
qualitative measurement for a quantitative resource allocation, 
sensitization to the needs and worries of some employees, 
enhanced public relations and a national reputation for 
innovation, a successful partnership with an educational 
institution and a government program. 

For the educational institution, the anticipated impacts 
were the development of a generic workplace basic skills 
curriculum, the establishment of a reputation as a provider of 
workplace basic skills programs, the enhancement of r.VC's 
reputation for innovation and flexibility, the cultivation of new 
partnerships and new sources of program funds, experience with 
responsive, community-based programming, experience with non- 
traditional delivery (ie. to shif tworkers) , 'evelopment of a 
model for project management and teaching in different 
environments, development of a model for creating and enhancing 
partnerships with business and industry. The unanticipated 
impacts were that JET served as a catalyst to implement a 
strategic plan set out by senior administration, resulted in a 
pool of instructors trained to undertake customized workplace 
programs, created a program of staff development for community- 
based programming, enticed Stelco workers to enroll in other AVC 
courses, provided AVC with a national profile in workplace 
literacy programming, and enhanced AVC's ability to demonstrate 
flexibility and responsiveness. 



13 



The anticipated impacts for the National Literacy 
Partnership Strategy partners included addressing the objectives 
of supporting innovative research-based literacy projects, 
encouraging workplace literacy projects, exploring the concept of 
partnerships between education, industry and government in 
addressing arlult literacy issues, encouraging the development of 
partnerships, supporting literacy initiatives in Alberta's post- 
secondary institutions, generating positive public relations, 
creating a demonstration project for example purposes, and 
accessing of federal monies by the private sector. The 
unanticipated impacts were the opportunities to learn a great 
number of workplace literacy specifics such as timing and 
recruitment, and partnership specifics such as flexibility and 
collaboration . 



Conclusion 

In addition to generating specific information about the 
nine objectives of the study and providing generalizations about 
the impacts of the JET program, this study directly and 
indirectly provided recommendations relative to workplace basic 
skills programs from each stakeholder to the others. Of 
particular interest is the inconguity between the measures of 
success pre-determined by the employer and employees and those 
espoused by the educational institution and the government 
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departments. The major recommendation that surfaces is that 
upgrading Canada's workforce may be most appropriately conducted 
while individuals are employed. 

In summation, it may be concluded that, with a relatively 
small budget, the Stelco JET program achieved a very large number 
of positive impacts, many planned and many unanticipated. 
Similarly, because of the large quantity, the quality and 
usability of the data generated, it has been concluded that the 
study of impact as a form of program evaluation may be a valuable 
and appropriate method of examining the benefits or outcomes of 
an educational program. 



A copy of the full report is available, at a slight cost to cover 
printing, from: 

Alberta Vocational College 

Research and Instructional Development 

10215 - 108 Street 

Edmonton, Alberta 

Canada T5J 1L6 



More information is available from: 

Dini Steyn (403) 422-4514 

AVC Program Marketing Manager 

Kathryn Chang Barker (403) 437-4491 
Kathryn Chang Consulting 
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